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Abstract 

We investigated the effects of synaptic depression on the macroscopic behavior 
of stochastic neural networks. Dynamical mean field equations were derived for 
such networks by taking the average of two stochastic variables: a firing state 
varialbe and a synaptic variable. In these equations, their average product is 
decoupled as the product of averaged them because the two stochastic variables 
are independent. We proved the independence of these two stochastic variables 
assuming that the synaptic weight Jij is of the order of 1/iV with respect to the 
number of neurons N. Using these equations, we derived macroscopic steady 
state equations for a network with uniform connections and a ring attractor 
network with Mexican hat type connectivity and investigated the stability of 
the steady state solutions. An oscillatory uniform state was observed in the 
network with uniform connections due to a Hopf instability. With the ring net- 
work, high-frequency perturbations were shown not to affect system stability. 
Two mechanisms destabilize the inhomogeneous steady state, leading two oscil- 
latory states. A Turing instability leads to a rotating bump state, while a Hopf 
instability leads to an oscillatory bump state, which was previous unreported. 
Various oscillatory states take place in a network with sjmaptic depression de- 
pending on the strength of the interneuron connections. 
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1 Introduction 



Neurophysiological experiments have shown that high-frequency inputs reduce 
the efficacy of signal transmission due to the exhaustion of neurotransmitters [T]. 
This phenomenon, caUed "synaptic depression" , provides a dynamic gain control 
by reducing presynaptic inputs [5J[nil5. The synaptic depression not only affects 
the activity of a single neuron but also the overall activity in neural networks [5]. 
To explore the effects of synaptic depression on the macroscopic behavior of 
stochastic neural networks, we analyzed a system based on a stochastic binary 
neural network model with synaptic depression by using mean-field theory. Pre- 
liminary results for the present work have been published elsewhere^. 

To observe the macroscopic behavior of the network, we reduced the stochas- 
tic neural network model with synaptic depression to microscopic dynamical 
mean field equations by taking the average of two stochastic variables, a firing 
state variable and a synaptic variable, over different realizations of stochastic 
spikes. If the two stochastic variables are independent, their average product 
can be decoupled as the product of their average. This decoupling enables a 
stochastic network model with synaptic depression to be reduced the closed 
form equations of their average. We demonstrated such independence for a 
non-frustrated system and derived microscopic dynamical mean field equations 
for a stochastic binary neural network model with synaptic depression. The de- 
rived equations coincide with those of an analog neural network with synaptic 
depression[a|3l|7]. 

Using these microscopic equations, we derived macroscopic steady state 
equations and analyzed the stability of macroscopic steady state solutions for 
two types of neural network : one with uniform connections and one with 
Mexican-hat type connections [5J [TOl E]- A network with uniform connections 
is the simplest type of network, for which the effect of synaptic depression has 
been studied [7 . A ring neural network with Mexican- hat type connections has 
with nonuniform connectivity and has been used as a model of the primary 
visual cortex and prefrontal cortex [TTJ [T^]. Although several researchers have 
reported that synaptic depression in the ring network leads to an oscillatory 
state, which is called a "rotating bump (RB)" state or a "traveling wave" state, 
little is known about the cause of the oscillation f8l fTOl [9] . 

We first discuss a network with uniform connections. Due to its simpleness, 
we can reduce the network to a model with only two components in which an 
oscillatory instability (a Hopf instability) can take place. This reduction enables 
us to easily understand why synaptic depression causes oscillatory instability. 
We first show that an oscillatory uniform (OU) state occurs due to the presence 
of synaptic depression. Although it has been reported that synaptic depression 
leads to an oscillatory state in a network with non- uniform connections [51 [TUl [5] . 
an oscillatory state has not been reported for a network with uniform connec- 
tions. 

Next, we show that, in a ring network with Mexican-hat type connections, 
the synaptic depression leads to three oscillatory states: the OU state, the RB 
state, and an oscillatory bump (OB) state, which was previously unreported. 
To investigate the mechanisms of steady state destabilization that lead to these 
oscillatory states, we analyzed the stability of the steady state solutions when 
there are frequency perturbations. In fj5l we show that high-frequency perturba- 
tions do not affect the system stability and that two mechanisms destabilize an 
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inhomogeneous steady state, leading to the two oscillatory states, i.e., the RB 
and the OB states. Analytical and numerical investigation of the ring network 
with synaptic depression revealed a phase diagram in which a wealth of states 
and multistable regimes can be discerned. 

2 Model 

We used a recurrent neural network with N neurons. The state of the i-th 
neuron at time t is represented by Si{t). The state takes either a resting state 
Si{t) = or a firing state Si{t) = 1. Each neuron follows a probabilistic dynamic: 

Prob[s,(t + l) = l] = gfi{h,{t)), (1) 
gf,{h,{t)) = i(l + tanh(/3/i,(i)), (2) 

where hi{= X^j^i Jij{'2xj{t)sj{t) — 1)) represents the total synaptic current ar- 
riving at neuron i, and l/f3{= T) is the level of noise due to the stochastic 
synaptic activity. At each time step, all neurons are updated in parallel. is 
a fixed synaptic weight from the j-th to the i-th neuron. Xj{t){Q < Xj{t) < 1) 
denotes the efficacy of signal transmission at the j-th neuron, which dynamically 
changes due to synaptic depression. Xj{t) is determined by the corresponding 
neuron state and itself at preceding time t ~ 1: 

Xj{t) = Xj(t - 1) + ^ ~ ^^^^ ~ '^^ - UscXjit - l)sj{t - 1). (3) 

In this model, synaptic connection Jij(t){— JijXj{t)) dynamically changes with 
the efficacy of signal transmission Xj (t) . The phenomenological model of synap- 
tic depression described by eq. ([3]) was proposed by several researchers [5J A 
schematic of this model is shown in Fig. [T] Information from one neuron (presy- 
naptic neuron) flows to another neuron (postsynaptic neuron) across a synapse. 
A synapse is a small gap separating neurons and consists of a presynaptic end- 
ing that contains neurotransmitters stored in synaptic vesicles, a postsynaptic 
ending containing receptor sites for neurotransmitters, and a synaptic cleft, or 
space, between the presynaptic and postsynaptic endings (Fig. [BJa)). An ac- 
tion potential cannot cross the synaptic cleft between neurons. Instead the 
nerve impulse is carried by chemicals called neurotransmitters (Fig. [IJb)). The 
cell receiving the nerve impulse (the postsynaptic neuron) has chemical-gated 
ion channels, called neuroreceptors, in its membrane. The presynaptic neurons 
exhaust neurotransmitters when they transmit signals. The efficacy of signal 
transmission at presynaptic neuron j at time t decreases by a certain fraction, 
UscXj{t — l){0 < Use < 1), after the firing of the presynaptic neuron, Sj{t — 1) = 1 
(Fig. [ijc)), and recovers with time constant t(t > 1), as shown in eq. ([3]). 

3 Mean field theory 

We propose a dynamical mean field theory for a stochastic binary neural network 
model with synaptic depression assuming that synaptic weight Jij is of the order 
of with respect to the number of neurons, N. 
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3.1 Microscopic mean field theory 

We derived microscopic dynamical mean field equations by first taking the noise 
average of the firing state variable at time t: 

{.s,{t + l))=gpi{h,m, (4) 

N 

(/i,(t)) =^J,,(2(x,(t)s,(i))-l). (5) 

Similarly, we consider the noise average of eq. ^ for the synaptic variable: 

{x,{t + 1)) = {x,{t)) + l^JMl _ U.^{x,{t)s,it)). (6) 

Evidently, Xi{t + r) and Si{t) are correlated when r > 0. However, equal time 
correlations between Si{t) and Xi{t) are of the order of that is, Xi{t) and 
Si{t) become "independent" when N ^ oo, as we will show below. 

Here, we define 6xi{t) — Xi{t) — {xi{t)) and 6si{t) — Si{t) — {si(t)). Substi- 
tuting eq. (HJ for {Sxi{t)6si{t)), we obtain 

{Sx,{t)ds,{t)) = {Sx,{t)gfi {h,{t - 1))). (7) 

Taylor expansion gives 

(M{t - 1)) = gp {{h,{t - 1))) + g'^ {{h,{t - 1))) S {h,{t - I)) + . . . . (8) 

Neglecting the higher order terms yields 

{5x,{t)5s,{t)) = g'p{{h,{t-l))){5h,{t-l)&x,{t)) (9) 

N 

= g'p {{K{t - l)))Y^J,,{5{xj{t - l)s,{t - l))6x,{t)). (10) 

We evaluate the order of the right hand side of eq. ([TU)) : 

{5{x,{t - l)s,{t - l))5x,it)) - {x,{t - l)){5s,{t - l)Sx,{t)) 

+ {sj{t - l)){Sxj{t - l)Sx,{t)) + {6xj{t - l)6sj{t - l)6x^{t)). (11) 

{S{xj{t — l)sj{t — l))6xi{t)) is of the order of 1/iV, because time-delayed cross- 
correlation is of the order of l/iV[13]. Since we have assumed Jy ^ 0{1/N), we 
obtain 

N 

J2 'h^i^ixjit - l)s,{t - l))6x.m ^ 0{llN) (12) 

and the equal-time correlations between Si{t) and Xi{t), {5xi{t)5si{t)) , disappear 
in the limit of large networks, N ^ oo: 

{x^{t)s,{t)) = {x,{t)){s^{t)). (13) 
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Taking advantage of the independence between Xi (t) and Si it) , we obtained 
the dynamical mean field equations for rriiit) and Xi{t): 

m,{t + l)=gp [y.^ .h, {2m,{t)X,{t) - 1) j , (14) 

X,{t + 1) = X,{t) + I^IM _ Us,Xdt)m,{t), (15) 

T 

where mi{t) = {si{t)) and Xi{t) = {xi{t)). These equations for the stochastic 
neural network model coincide with those for an analog neural network with 
synaptic depression [7]. The steady state equation for noise average Xj = Xj{oo) 
is 

= T-^ , 7 = tC/so, (16) 

1 + ^ruj 

which is a finite temperature version obtained in the T — case[14j. Finally, 
we obtain the microscopic steady state equation for mi{— mi{oo) for a network 
with synaptic depression: 



According to eq. ([T7]), steady state depends on 7(= T[/se)[7]. 
3.2 Stability analysis 

To examine the stability of the steady state obtained with eq. (1171) , we consider 
small deviations around a fixed point [71 151 [T5]: 

m,{t) =m, + 6m,{t), X^{t) = X, + 5X,{t), (18) 

where Xi = j^^^- We linearize eqs. ((Ti)) and (IT51) about the steady state 
solution. Neglecting the higher order terms, we obtain 

h, = J,j {2mjXj - 1) . (20) 

Similarly, 

SX,{t + 1) = ~UseX,Sfm{t) + (^1 - ^ - Uscm.^ 5X,{t). (21) 

Next we calculate the partial differential coefficients ^^q}-^''^ and of eq. 

in more detail. 

dgiiihi) _ dgpih,) dh, ^^2) 



drrij dhi drrij 

/3J,,X, (l-(tanh/3(/i,))2) (23) 
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Substituting the steady state equation. ([T7|) into eq. we obtain 

dgi3{h 



= 4/3 J,,X, (m, - m^) . (24) 



Similarly, we have 



AI3J,.jm.j (mj - mf) . (25) 



Substituting eqs. (HH) and into eq. (HH) yields 

5vi,{t + 1) = ^ 4/3 (to^ - ml) {Xj5mj {t) + irijdXj [t)) . (26) 

From the relations for the coefficients of eqs. (|2ip and (^5]) . we obtain the 
Jacobian matrix for the systemfSJ. The Jacobian matrix K has a size of 2N x 2A^ 
with matrix elements as follows. 

"W™ K'U' \5X,{t + l))-'' \6X,{t))^ ^''^ 

K^m = 4/3 J,, (m, - )X, , K^x = 4/3 J., {m, - m\)vfi, , (28) 



= t^scX, , iCix ^ 5,^. (^1 _ _ c/,^^^. , (29) 

where 1 < i,j < N and (5^^ is the Kronecker delta. If the Jacobian matrix has 
eigenvalues of 1 or less, the steady state solution is stable. 



4 Network with uniform connections 

In a network with uniform connections, 

Jy = Jo/N. (30) 

4.1 Macroscopic steady state equations 

We derived macroscopic steady state equations for a network with homogeneous 
connectivity by using the microscopic mean field equations p4p and (llSp . Given 
the symmetry of the synaptic weights in eq. pOI) , we can set the noise average 



ji— 

condition into eq. (jl7p . we obtain a macroscopic steady state equation 



of each neuron, m.^, to nii ~ mo, where itlq = ;^X]i=i"^i- Substituting this 



Eq. gives the homogeneous steady state solution. 
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4.2 Stability analysis 

To examine the stability of the homogeneous steady state solution obtained 
using eq. (I32L namely nii = mo and Xi ~ Xq, we consider small deviations 
around a fixed point [71 l8l [T5]: 

m,{t) = 7710 + Sm,it), X,{t) =Xo + 5X,{t). (33) 

Substituting eqs. ((30)) and (p3t into eqs. (fT9)) and ((26t . we obtain 

1 ^ 

5m,(i+l) = 4/3Jo(mo-™o)^I](^o'5^"j W+"^o^^, W) • (34) 



5X,{t + 1) = -C/,eXo<5m,(t) C/semo j ^^»(i). (35) 

Since J,j consists of 0-th the Fourier component of Jo, discrete Fourier transform 
analysis can be use to diagonalize Jacobian matrix K (eq. (P7)) ). We therefore 
compute the Fourier series for eqs. ([34]) and ([35]) . The Fourier coefficients of 
Snii (t) and 6Xi {t) are given by 

1 ^ - ■ 1 ^ 

,5mfe(t) = -^<5m,(0e^^, ,5Xfc(i) = - ^ 5X,(i)e^^ , (36) 

1=1 1=1 

where i is the standard imaginary unit with the property i^ = —1. We then 
write 5mi{t) and 5Xi{t) in Fourier series form: 



5m,{t)= ^ 5rhk{t)e^^, 5X,{t) = ^ 5Xk{t)e^^ . (37) 
Substituting eq. ([57)) for eqs. (IMl) and ([55)) . we obtain 

f-l ^^^^^^^ f-l ^ A 

^ (5mfe(i+l)e^^ =4/3Jo(too -go) [XoSmi{t) + modXi{t)'^ 



e « , 

fe=-4 i=-f " 

(38) 

4/?Jo (mo - go) (^o^"io(t) + moSXoit)) , (39) 



A, 2 ^ 2 

-i7,e 51 ('JmKO^HO +™i(i)'5^;'(i)) e , (40) 
M'=-f 
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where qk = " • We use the fohowing equation in the hmit 

of cx) to integrate the right side of eq. ([55]) with respect to j: 



-y 



'1 (/ = o) 



Since Fourier components are orthonormal, we can equate the coefficients of 
the Fourier components on the left and right sides. From the relations for the 
coefficients of eqs. ([39)) and (|40|) . we obtain the Jacobian matrix for the system 
in Fourier space, H^^: 



fc = 



[sXo{t + l)) - [sXoit)) ^ ' 



f 4:13 Jo (mo - go) ^0 4/3 Jo (mo - go) ™o\ f 5mo(t) 



(43) 



k |> 1 



(5mk{t + l)\ ^ rrkk ( ^rnk{t)\ , . 

^v5Xfc(t + l)y ^<5Xfe(t)y ^^^^ 

^ / ^ /<5mfc(t)\ , . 

^-[/scXo 1 - i - C/,cmoy ^^^(t)^ ■ ^ ' 

This form makes it easy to analyze the stability of a steady state since, for 
any fc, the time evolution of each equation pair {Srhkit) and SXk{t)) decouples 
from all other equation pairs. Eigenvalue Xk of H'^'^ determines whether a steady 
state solution is stable or not. The solution is stable if it is 1 or less. The values 
for the eigenvalues, Xk{7^ 0), are given by Afc = and (l — — Uscmo). Since 
1 < T, < ?7sc < 1, < mo < 1, I Afe |< 1, and rhk, Xk ^ a.s t oo. We 
consider the stability to homogeneous perturbations Sttiq and 6Xq, and obtain 
eigenvalue Ao of iJ*'": 



4/3Jo (mo - go) +( 1 - - - Uscrhf) 



T 

±<i ( 4/3 Jo (mo - go) + ( 1 - ^ - UscTho] - 4 U(3Jo (mo - go) [l - ^ 



(46) 



We find that two types of linear instability of the stationary uniform state are 
possible |16]. For | A^ |> 1 and Im(Ai|^) — 0, homogeneous perturbations Srho 
and 6X0 cause firing-rate instability. If | A^ | is more than 1 and Im(A^) 7^ 0, 
homogeneous perturbations Srho and 6X0 yield a Hopf instability. This results 
in growing oscillations and instability. A small-amplitude limit-cycle periodic 
solution exists near the steady state solution. 
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4.3 Results 



To evaluate the affect of synaptic depression on a network with uniform con- 
nections, we investigated the steady state solutions and their stability in the 
network. We obtained the steady state solutions to eq. (|32|) by solving 

M = mo, (47) 

M^Ul + tanh/3Jo ( - A) ■ (48) 

Graphical solutions to eqs. (j47l) and are shown in Fig. Ufa) for T — 0.3 
(dot-dashed line) and T — 0.8 (dashed hue) when 7 — rC/gc = 0.35. Regardless 
of /3Jo, the line for eq. (|47)) passes through a point (23^, 0.5) as shown in Fig. 
[2fa). When the level of noise was low (T = 0.3), eq. had three fixed points. 
Since two of them were attracting and the other was repelling, we found that 
the network with uniform connections had a bistable region at low and high 
Too values in the case of low noise. We defined a ferromagnetic (F) state as 
bistability in too[T7]. When the level of noise was high (T = 0.8), the network 
with uniform connections had a monostable state for the mo values. It is called 
a "paramagnetic (P)" state[T7]. When 7 > 1, > 1 and the network had 
only a P state. 

The results of steady state solutions obtained using eq. (15^ and stability 
analysis for t = 2 with a fixed degree of synaptic depression (7 = 0.35) are 
shown in Fig. [H^b). For 7 = 0.35 and r = 2, the equilibrium point given by eq. 

was stable. The solid lines in Fig. ^h) denote stable solutions, and the 
dashed line denotes an unstable solution. A saddle node bifurcation occurred 
at Tc = 0.36 since | |> 1 and Im(Aj) = (firing rate instability). Hence, the 
F state was stable for a low noise level (T < Tc), while the P state was stable 
for a high noise level (T > Tc). Using eq. we found that all solutions 

with Too < 0.5 were stable. This means that synaptic depression stabilizes a 
low firing-rate state in a network with uniform connections. 

4.3.1 Hopf instability of homogeneous solution 

We investigate the effect of the time constant r on the stability of steady state 
solutions. We set 7 to 0.35 and r to 2 and 100. The steady state solutions 
obtained using eq. ([5^ and stability analysis are shown in Fig. EJa) for r = 2 
and in Fig. ^h) for r — 100. The simulation results are represented by the 
squares. The number of neurons was 10"^, and the initial conditions for the firing 
state variables and synaptic variables were 5^(0) = 1 and 2:^(0) — 1 for all i and 
Si{0) = and Xi{0) = 1 for all i. Since 7 — 0.35 in both cases, the same steady 
state solutions are the same. We found that, near the transition point between 
the P and F states, a fixed point (toq > 0.5) was stable for r = 2 (Fig. |3l^a)), but 
unstable for r = 100 (Fig. |3i;b)). Since | |> 1 and Im(A^) ^ at toq = 0.865 
and T = 0.353, as shown in Figs. Sljb) and (c), a Hopf bifurcation occurred, and 
there was an OU state near T = 0.353. Fig. [5]shows rho{t){— X]f^"^i(^)) ^^id 
Xo{t){— X)r^i(O) for the OU state, respectively given by eqs. (fH)) and 
(A^ = 10'^). When T was more than 0.353, the OU state was unstable, and the 
macroscopic property of the network changed to the P state. These results are 
consistent with the simulation. The phase diagram of the stability for mo > 0.5 
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shown in Fig. [3{c) reveals that the state tended to be unstable when the time 
constant of the synaptic variable increased. The unstable region expanded until 
T « 50, and then it settled in an approximately constant region. These results 
show that the stability of the solution strongly depends on the time constant, 

T. 

Finally, we discuss the oscillatory instability in the neural network with 
uniform connections in the presence of synaptic depression. A neural network 
with synaptic depression can be considered an activator-inhibitor system in 
which firing rate m is an activator and synaptic variable X is an inhibitor. 
This is because inhibitor X activated by m and inhibits activator production 
since total synaptic input h decreases as X increases. In the activator-inhibitor 
system, an oscillatory instability (a Hopf instability) can take place 1 181 [TO] . 
We found that the oscillatory uniform state remained for a sufficiently long 
time constant in a neural network with uniform connections in the presence of 
synaptic depression. 



5 Ring Attractor Network with Mexican-hat type 
connectivity 

For a network with uniform connections, we found that the homogeneous state 
solution is unstable and that oscillatory uniform state remains for a sufficiently 
long time constant. 

Let us turn now to a ring attractor neural network with Mexican-hat type 
connectivity. In this network, N neurons are located on a one-dimensional ring 
parameterized with 9 G [0,2tt). Neuron i is labeled using angle 0.i{= ^ — 
The synaptic weight is 

J^, = ^ + ^cos2i0,-0,), (49) 

where Jq is a uniform interaction, and Ji is a lateral inhibition interaction 
[TTl [20j [T7].The model with Ji = reduces to a network with uniform connec- 
tions. For Jq — 0, the network is a balanced network with Mexican-hat type 
connectivity, like that studied by York et a/[5]. 



5.1 Macroscopic steady state equation 



Using the microscopic mean field equations (1141) and (fTSj) . we obtain macro- 
scopic steady state equations for the ring network with synaptic depression. 
Substituting eq. into the microscopic steady state equation (fT7|) , we obtain 
a microscopic equation: 

m = gpihi), r, = — , (50) 
1 + ■yrrii 

h = Joro + Ji (^^^^i^^cos(20O + ^^^^-^f^sin(20,)) , (51) 

where 

N N N 

i—1 2—1 i—1 
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The renormalized output of the i-th neuron due to the synaptic depression is 
ri = TOi/(l +"/mi). We can obtain the following self-consistent equation for the 
macroscopic steady state in the limit iV — >■ oo, 

m = i r d92r{9) - 1, ri = ^ r de2r{9)e-^''^ , r^i ^ ^ de2r{9)e^'^ , (53) 



^ .g,(Joro + Jl(rie-^'^ + r-ie^'^)) 
"^"^ l + 7ff^(Joro + Jl(rie-2i« + r_ie2i«))- ^ 

The order parameters given by eq. (I54p are used to calculate the firing rate 
(mo — ^ /_^7r d9m{9)) and the 1-st order Fourier component of m{9) (mi — 

T /-^ d9m{9)e~^^^). This component, mi, indicates the degree of activity lo- 
calization. 

There are two types of solutions to eq. ([5^ . One is a homogeneous solution 
with mi = 0, and the second one is a bump solution with mi ^ 0, which is 
inhomogeneous . 

5.2 Stability analysis 

We investigated the stability of the steady state solution given by eq. (j54p for 
the ring network as done for a network with uniform connections [71 151 115|. To 
examine the stability of the steady state solution, rrii and Xi, obtained using 
the eq. ([5^ . we considered small deviations around a fixed point in eq. ([5^ : 



m,{t) = m, + Sm,it), X,{t) =X, + SX,{t). (55) 

Since Jij consists of the 0-th Fourier component of Jq and the 1-st Fourier 
component of Ji in the ring network, discrete Fourier transform analysis can be 
a used to diagonalize Jacobian matrix K (eq. ^7} ) for the ring network as well 
as for a network with uniform connections. We hence write 6mi{t) and 6Xi{t) 
in Fourier series form in a way similar to that used in i i4.2l Substituting eqs. 
(|57| and (gSl) and 61, = ^ - f into eqs. dH]) and (HH), we obtain 

N 



Smkit + l)e2"=^- ^J^J2 ^^^-^^ + 4/3 Ji cos 2{9, - 9,)) 

=-f 

«_i 



k' ,Ll' = - 



-1 



{rhk' - Qk') 



4/3 Joe''"' J2 mm^iit) 



' 2 
« _1 



+2^Jie2'('='+i)«' Y Mt)X^ii^r){t)+2pJie-''<''~')'^ ^ m,(<)X_(,+i) (i) 



(57) 
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2iW, 



Use [5Mt)Xi'{t)+rhi{t)5Xi,{t))e^'<'+'y^, (58) 



where we use the following equation in the limit of — cx3 to integrate the 
right side of eq. ([55)) with respect to 6j , 



Since Fourier components are orthonormal, we can equate the coefficients of 
the Fourier components on the left and right sides. From the relations for the 
coefficients in eqs. ([55)) and ([57)) . we can obtain the Jacobian matrix for the 
system in Fourier space, H. The matrix has a size of 2N x 2iV, with matrix 
elements 

^Am = Wo("ife - qk)X-i + 2/3Ji(mfe„i - qk-i)X-(^i-i) + 2/3Ji(mfe+i - gfc+i)X_(i+i), (61) 
^AX = 4/3Jo(mfc - qk)m-i + 2/3Ji(mfc-i - (7fc-i)m_(;_i) + 2l3Ji{mk+i - gfc+i)rn_(i+i), (62) 

^Jtrh = -UsA-u (63) 

= Ski (l - 7) - (64) 

where — ^ £ T"-*^' ^-'^'^ ^'^^ Kronecker delta. If the Jacobian 

matrix has eigenvalues of 1 or less, the steady solution is stable. 

First, we consider the stability of homogeneous steady state {rhk = and 
Xk = {k ^ 0)), which can be analytically analyzed as shown below[F. If the 
network has a homogeneous steady solution, Xk = 0, rhk = (fc 7^ 0), and we 
have 

^'■'=(0 0) (MO- (65) 

This equation shows that the time evolution of each equation pair (Srhk (t) and 
dXk{t)) decouples from all other equation pairs. The Jacobian matrix for the 
ring network is therefore as easy to analyze as that for a network with uniform 
connections. The Jacobian matrix thus reduces to the following matrices: 



fc 



^00 ^ /4/3 Jo (too - go) -^0 4/3 Jo (mo - go) mo\ , , 

^ -UseXo 1-i-C/seTOo ' ^ ' 
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A: = ±1 



^11 ^ /2f3Ji{mo - qo) Xq 2/3Ji (mo - (Jo) mo\ ^g^^ 

V -UscXo 1 - 7 - C/sc'Tlo / 

H-^-^ = ff" (68) 

k \> 1 

Since the eigenvalue At of TJ'^'^d fc |> 1) is less than 1, 5mfc, SXk — > as 
t — ^ 00. We consider the stability under perturbations Sttiq, SXq, Smi, and 
SXi. The stability under perturbations (5m_i and 6X^i is identical to that 
under perturbations Srhi and SXi . The eigenvalue of i?*^" is given by eq. (|46p . 
Next, we obtain eigenvalue of H^^: 



A 



± 



1 



1 ~ 2 



2^Ji (toq - go) +( 1 - - - C^scTOo 



T 



±<i ( 2/3 Ji (rno - go) + ( 1 - i - C/scmo j j - 4 (^2/371 (mo - go) U - ^ 



(70) 



The stability of the homogeneous steady state solution is determined by eqs. 
(Uni) and (fTO)) . There are four types for linear instability of the homogeneous 
state: (1) firing-rate instability (| Ag |> l,Im(Ao ) = 0, | Af |< 1), (2) Hopf 
instability (| |> l,Im(A;^) ^ 0, \ Xf |< 1), (3) Turing instability (| A^ |< 1, 
I X^ |> 1, Im(Af) = 0), and (4) Turing-Hopf instability (| AJ |< 1, | Af |> 1, 
Im(Af) ^ 0)[I5]. If there is a Turing-Hopf instability, a spatially homogeneous 
steady state solution is unstable and, spatial periodic patterns evolve. 

In contrast to the homogeneous steady state solutions, it is difhcult to an- 
alyze the stability of inhomogeneous steady state solutions since the time evo- 
lution of each equation pair {rhk{t) and Xk{t)) is coupled with other equation 
pairs. We hence have to take into account stability under frequency perturba- 
tions 5rhk and 5Xk {—N/2 < k,l < N/2 — 1) since the highest Fourier component 
is m±Ar/2, X±]\f/2 in a network with TV by sampling theorem. Here we study 
how frequency perturbations 6rhk and SXk i—N/2 < k,l < N/2 — 1) affect 
the stability of the inhomogeneous steady state solutions by considering the 
eigenvector of the Jacobian matrix (eq. (jHOl) ). 



5.3 Results 

5.3.1 Oscillatory states 

Our investigation of the stability of the steady state solution given by eq. ([5^ 
for the ring network revealed six states in the network. Three arc homogeneous 
and were also found in a network with uniform connections: a ferromagnetic (F) 
state, a paramagnetic (P) state, and an oscillatory uniform (OU) state. The OU 
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state occurs in a way similar to that described in i i4.3.1l fFig. U). The other three 
states are inhomogeneous: a bump (B) state, a rotating bump (RB) state[Sl[n], 
and an oscillatory bump (OB) state, as shown in Fig. [S] The B state can be 
obtained using a self-consistent equation, eq. while the other two cannot 

because they are dynamic states resulting from the destabilization of steady 
states. We thus obtained them by using the dynamical mean field equations, 
(HH) and (IT5|) with N — 10**. The firing rate m^, average of the synaptic variable 
Xi, and synaptic weight between i-th neuron with the preferred orientation 
9i = and the other neurons are shown in Fig. [5] for the three inhomogeneous 
states. 

First, we discuss the behavior of the three inhomogeneous states in a ring 
network with synaptic depression. The B state is formed by a subset of the 
neurons firing in a self- reinforcing manner, causing localized activity (Fig. [SJa)), 
similar to the B state in the network without synaptic depression pH \7U[ 15] . In 
the B state, the firing rates are high, while the averages of the synaptic variables 
are low (Figs. [6ja) and (b)). The synaptic depression thus reduces the excitatory 
localized interaction and reduces the presynaptic inputs of the activated neuron, 
as shown in Fig. Hfc). In the RB state, a localized bump of activity propagates 
around the ring network, leaving a wake of replenishing synaptic resource, as 
shown in Figs. El (d) and (e)[H]. In this example, the profile is moving to the 
right. As a result, the synaptic weights dynamically changed in the RB state 
(Fig. IlKf)). In the OB state, which is first reported here, the bump state activity 
moved up and down around the neurons with the firing rates that were high, 
as shown in Figs. [Hg) and (h). Unlike in the RB state, the moves are tiny in 
the OB state (Figs. EUg), (h) and (i)). The occurrence of the two oscihatory 
states from the B state implies that there are two mechanisms destabilizing the 
B state. 

Next, we provide evidence that the simulation results coincide with the dy- 
namic solution obtained using eqs dH]) and for the B state (/? Jq = 0, /3 Ji = 
10) and the RB state {13 Jq = 0, /3Ji = 6.5). Figs, ^a) and (b) show raster plots 
of neuron activities obtained by numerical simulation with N = 10**. The solid 
lines represents the dynamic solution obtained using eqs. (fill) and dTSl) . In the 
B state, the localized position fluctuated and moved around the ring network 
since the B state is stable anywhere in the ring (Fig. [7Ka))|2D]. In the RB state, 
the bumps propagated stably, i.e., there was a traveling wave (Fig. [71[b))[Hl IH]- 

In short, we found six states of activity in the ring network with synaptic 
depression: paramagnetic, ferromagnetic, bump, oscillatory uniform, rotating 
bump and oscillatory bump. 

5.3.2 Hopf and Turing instability of inhomogeneous solution 

We considered the stability of the inhomogeneous steady state solution (the B 
state) in order to identify the destabilization mechanisms leading to the two 
inhomogeneous oscillatory states, the RB and OB states. To analyze the sta- 
bility of a network with N = 10^, we computed the eigenvalues of the Jacobian 
matrix H (eq. (pO]) . 2000 x 2000) for frequency perturbations Srhk and SXk 
(—5 X 10^ < fc < 5 X 10^ — 1). Note that the highest Fourier components 
are m^^xio'^ and X-1-5X102 in a network with N = 10'^, as determined by the 
sampling theorem. 

To begin our analysis, we consider a network with a fixed degree of synaptic 
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depression (7 — 1.5) and a time constant (r — 3), in which the RB state occurs 
near the transition point between the P and B states. Fig. [Sl^a) shows how 
the ampHtude of locahzed activity, rhi, depends on pji for /3Jo — 0. The sohd 
and dashed Unes represent stable and unstable solutions to eq. Fig. MJ^) 

shows the distribution of eigenvalues for Jacobian matrix H. There are two 
mechanisms of destabilization that lead to the RB state. First, as /3Jo crosses 
4.5 from below, the P state becomes unstable and a Turing-Hopf instability 
leads to the RB state occurring because | Aq |< 1, | Af |> 1 and Im(Af ) 7^ 0. 

Next, we show that as fSJi crosses 8 from above, the B state becomes unstable 
and a Turing instability leads to the RB state. Fig. [DJb) shows that there are 
eigenvalues continuously distributed between A = 0.15 and 0.65 that do not 
contribute to destabilization and that there are a few eigenvalues greater than 
1 that do. Figs. [5Jc) and (d) show the eigenvector of the largest eigenvalue for 
/3Jo = and /3Ji ~ 6.5. It is indicated by the x mark in Fig. ^h). We see from 
Figs. Eljc) and (d) that their eigenvectors mainly consist of Srhi, 6X1, Srh-i, 
and 6X^1 and not Sttiq or 6Xq. In addition, they do not have an imaginary 
part. We hence found that the B state was unstable and that the RB state 
occurred due to a Turing instability. 

Next, we consider a ring network with 7 = 2.5 and t = 3, in which the B state 
is unstable and the OB state occurs at the transition point (/3 Jq = 2.47) between 
the B and P states. As /3Jo crosses 2.47 from below, the OB state becomes 
unstable and the P state occurs. Fig. [Ha) shows how the amplitude of localized 
activity ihi depends on pjo for /3Ji = 20. Fig. MJ^) shows the distribution of 
eigenvalues for Jacobian matrix H. Only a few eigenvalues are greater than 
1, as they were in Fig. [HUb). Figs. M,c) and (d) show the eigenvectors of the 
largest eigenvalue for /3 Jq = 2.47 and (3Ji = 20 indicated by the x mark in Fig. 
[9Ub), which indicates that the eigenvectors mainly consisted of Srho and SXq. 
Furthermore, they had an imaginary part. The B state was thus unstable, and 
a Hopf bifurcation led to the OB state for /3Jo = 2.47. 

These results clearly show that there were two mechanisms destabilizing the 
B state, i.e., a Turing instability and a Hopf instability, which led to the RB and 
the OB states. Moreover, we found that there were few eigenvalues that were 
larger than 1 and thus affected the stability of the system, as shown in Figs. 
[5] and m and that their eigenvectors consisted of only low-frequency Fourier 
components. 

5.3.3 Phase diagram 

We investigated how neuron interactions affect the macroscopic states of net- 
works by changing the strength of the uniform connections ( Jq) and the lateral- 
inhibitory connections (Ji). Fig. 1101 shows the phase diagrams identified in the 
(/3Jo, (3Ji) plane with a fixed degree of synaptic depression (7 = {0, 0.4, 1.5, 2.5}). 
To analyze the stability of the steady state solutions in a network with N — 10"^, 
we computed the eigenvalues of Jacobian matrix H for only low-frequency per- 
turbations, namely Srhk and SXk (—5 x 10 < k < 5 x 10 — 1), since high- 
frequency perturbations do not affect the stability of the system when there 
is weak lateral-inhibitory interaction (Ji) (See Appendix). The instability of 
an oscillatory state was numerically investigated using eqs. (jl4p and (fT5|) with 
N = 10^. 

To begin with, we describe the behavior of a network with nondepressed 
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synapses (7 — 0). Fig. [TUfa) shows the phase diagram for 7 — 0[5(T. The 
relative strength of /3 Jq and (3Ji determines the network state. The F or B states 
become stable once /3Jo and (3Ji exceed certain thresholds. Between these two 
states, there are bistable regions where both F and B states are locally stable 
(F+B). If both /3Jo and /?Ji are small, a P state is stable. 

In the presence of weak synaptic depression (7 = 0.4, r = 3) (Fig. [TUTb)). 
the P region expanded and the bistable regions shrunk since synaptic depres- 
sion effectively reduced the lateral-inhibitory ( Ji) interaction. Bistable regions, 
where both P and B states were locally stable (P-l-B), developed. As the degree 
of synaptic depression increased (7 — 1.5, r = 3), the RB state became stable 
near the transition point between the P and B states, as shown in Fig. [TOF c'). 
Since 7 > 1, the F state was unstable. 

In the presence of strong synaptic depression (7 = 2.5, r = 3) (Fig. fTUfd)). 
the RB region expanded and the P-l-B region shrunk. Bistable regions, where 
both the P and RB states are locally stable (P-I-RB), developed. The B state 
was unstable, and an OB region developed at the transition between the B+P 
and P states (Fig. ini[g)^(i)). Otherwise, the OB state was unstable, and the P 
state was stable. 

We have shown that for a sufficiently strong degree of synaptic depression, 
the B state is unstable and the oscillatory states (RB and OB) occur near the 
transition of the B and P states. Sufficiently strong lateral inhibition interaction 
(Ji) leads to a Turing instability and the RB state occurs. Sufficiently strong 
uniform inhibition interaction Jq leads to a Hopf instability, leading to the OB 
state. 

5.4 Summary 

In the ring attractor network with synaptic depression, there are homogeneous 
steady states (F, P) and an inhomogeneous steady state (B). We have shown 
that, depending on the strength of the interneuron connections, instability in 
these states leads to three oscillatory states: oscillatory uniform (OU), rotating 
bump (RB), and oscillatory bump (OB). 

We summarize how the interactions between neurons affect the stabilities of 
the steady state solutions by changing the strengths of the uniform connections 
(Jo) and the lateral-inhibitory connections ( Ji). Fig. [TT] shows a schematic view 
of the stability analysis for a ring network with synaptic depression. First, for 
sufficiently weak uniform connections and a sufficiently long time constant (r), 
the homogeneous steady state solution, which has a high firing rate was unstable 
near the transition point between the P and F states. An OU state then devel- 
oped due to a Hopf bifurcation, as explained in ii5.3.1l (Figs. fUT a')— »(i)). Note 
that the OU state developed in the presence of lateral-inhibitory connections 
as well as in their absence. Next, as the strength of the uniform connections 
increased, the inhomogeneous steady state solution became unstable near the 
transition point between the B and P states (Fig. The OB state then de- 
veloped because of a Hopf instability, as explained in i )5.3.2l fFigs. ITlTbWfiii')). 
In the OB state, bump state activity moved up and down around the most acti- 
vated neuron (Figs. ini[g)-(i)). Finally, homogeneous state P and inhomogeneous 
state B became unstable near the transition point between them (Fig. [5Ja)), and 
the RB state developed, as explained in H5.3.2I (Figs, ini^d)-(f)). As the strength 
of the lateral-inhibitory connections increased, the homogeneous steady state 
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solution became unstable, and a RB state developed because of a Turing-Hopf 
instability (Figs. mT aWfii')). As the strength of the lateral-inhibitory connec- 
tions decreased, the inhomogeneous steady state solution became unstable and 
an RB state developed due to a Turing instability (Figs. HHb)— !>(ii)). 

These results show that medium-strength uniform connections cause a Hopf 
instability near the transition point and up-and-down movement of the firing 
rates and the average synaptic variables, and that medium-strength lateral- 
inhibitory connections cause a Turing instability near the transition point and 
propagation of a localized bump of activity around the ring network. In sum, 
various oscillatory states take place depending on the strength of the interneuron 
connections in a ring network with synaptic depression. 

6 Conclusion 

We have explored the macroscopic properties of two types of stochastic binary 
neural networks with synaptic depression: a network with homogeneous con- 
nectivity and a ring attractor network with Mexican-hat type connectivity. We 
proposed a dynamical mean field theory for a stochastic binary neural network 
model with synaptic depression assuming that synaptic weight Jij is of the order 
of with respect to the number of neurons {N). Using microscopic mean 
field equations, we derived macroscopic steady state equations for these net- 
works and investigated the stability of the steady state solutions obtained. The 
results coincided with those from simulation. We conclude that the presence 
of synaptic depression leads to oscillatory instability and that various oscilla- 
tory states take place depending on the strength of the interneuron connections. 
Synaptic depression thus causes a diversity of dynamic states in large networks 
of spiking neuron. 

We focused only on non-frustrated systems, in which ~ 0{1/N). A 
further direction of this study will be to extend the microscopic dynamical 
mean field theory to frustrated systems, such as the Sherrington and Kirkpatrick 
model ^21]. 

7 Appendix: Dimensionality reduction for sta- 
bility analysis 

In Section 15.3.21 we showed that few eigenvalues consisting of low-frequency 
perturbations affected the stability of the system. This means that only low- 
frequency perturbations affect the stability of the system. In this section, we 
compare the eigenvalues of the Jacobian matrix in Fourier space (eq. (1601) ) for 
high and low frequency perturbations with those for low-frequency perturba- 
tions, as indicated in Figs [12] and [TS] Fig. [T2jb) shows the distributions of 
eigenvalues for the Jacobian matrix (eq. I60p for perturbations Srhk and 5Xk 
(—5 X 10^ < fc < 5 X 10^ — 1) in a ring network with 7 = 1.5 and r = 3 as 
well as Fig. |Hl[b). Figs. [T^d) and (e) show those for low- frequency perturba- 
tions, namely 6rhk and 6Xk (~5 x 10 < fc < 5 x 10 — 1), and 5rhk and 5Xk 
(—5 < fc < 5 — 1) respectively. Fig. 1131 shows a comparison for a ring network 
with 7 = 2.5 and r = 3 in the same way. 
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Although we did not approximate the distributions in Fig. [T^] (b) as a 
whole by using the distributions shown in Figs. [T^] (d), we found that the 
maximum eigenvalue of the Jacobian matrix for perturbations Srhk and SXk 
(— 5x 10 < k < 5x 10 — 1) coincided with that of the Jacobian matrix for 
perturbations 6rhk and 6Xk (—5 x 10^ < fc < 5 x 10^ — 1). However, the results 
of stability analysis for perturbations Srhk and SXk (~5 < k < 5 — 1) and 
those for perturbations Srhk and 5Xk (~5 x 10^ < fc < 5 x lO'^ — 1) differed in 
the maximum eigenvalue of the Jacobian matrix because the lateral-inhibitory 
interaction (Ji) was relatively strong (Figs. [111(b), and(f)). In contrast to the 
stability analysis for Srhk and SXk (—5 x 10 < fc < 5 x 10 — 1), we obtained 
different results for the stability analysis. We obtained the same results of 
stability analysis, as shown in Fig. [T31 

These results show that high-frequency perturbations irik and Xk did not 
affect the stability of inhomogeneous steady state solutions because the lateral- 
inhibitory interaction (Ji) was relatively weak. Therefore, when there is weak 
lateral-inhibitory interaction, we can reduce the dimensions for the stability 
analysis since we do not need to take into account high-frequency perturbations. 
This dimensionality reduction enables rapid analysis of the stability of steady 
state solutions for a ring network with synaptic depression. 
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Figure 1: Schematic of phenomenological model of synaptic depression 
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Figure 2: (a) Graphical solutions to eq. ((5^. (b) Stable and unstable solutions 
to eq. dSl]) for 7 = 0.35, r = 2. 
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Figure 3: Squares in figures (a) and (b) represent average m numerically ob- 
tained by simulation with N — 10"^ . Solid lines denote stable solutions, and 
dashed lines denote unstable solutions, (c) Phase diagram for stability of 
mo > 0.5 state. 
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Figure 4: Results of stability analysis for rho > 0.5 and 0.34 < T < 0.362. (a) 
Firing rate rho state: solid line denotes stable solutions and dashed lines denote 
unstable solutions, (b) Absolute value of A^. (c)Image part of Xf. 
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Figure 5: (a) Firing rate rho{t) and (b) average of synaptic efficacy Xo{t) cor- 
responding to OU state. 
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Figure 6: Profiles for tliree inlromogeneous states (B, RB, OB), (a), (d), (g) Fir- 
ing rate mj. (b), (e), (h) Average of synaptic variable Xi. (c), (f), (i) Synaptic 
weight Jij (t) = JijXj {t) between i-th neuron with preferred orientation 6i = 
and other neurons. 

(a) B state (y=1.5, t=3, |3J^j=0, PJj=10) 
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(b) RB state (y-1.5, t=3, ^J=0, =6.5) 




time 



Figure 7: Raster plots of neuron activity for numerical simulation (A'^ ~ 10"*). 
Solid lines denote temporal behaviors of bump position </>. (a) B state, (b) RB 
state. 
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(a) j = l.5,T = 3,0Jo = O (c) /3Jo = 0,/3Ji = 6.5 
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Figure 8: (a) Stable (solid lines) and unstable solutions (dashed line) to ampli- 
tude of localized activity rhi . (b) Distribution of eigenvalues for Jacobian matrix 
with size of 2000 x 2000. (c), (d) Eigenvector with largest eigenvalue, 1.1 
for PJq — and pji = 6.5, as shown by x mar kin (b). 




Figure 9: (a) Stable (solid lines) and unstable solutions (dashed line) to ampli- 
tude of localized activity rhi . (b) Distribution of eigenvalues for Jacobian matrix 
iJ lpO)) with size of 2000 x 2000. (c), (d) Eigenvector with largest eigenvalue, 
1.04 for /3Jo = 2.47 and pji = 20, as shown x in (b). 
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Figure 10: Phase diagrams with fixed 7 in interaction of (/3Jo, PJi) plane for 
7 = {0,0.4,1.5,2.5}. For (b) to (d), r = 3. P : monostabihty in mo, and 
mi = 0. F: bistability in mo, and mi = 0. B: mi ^ 0. RB : mi 7^ 0. OB 
state occurs on transition line between P+B state and P state with 7 = 2.5 and 
T = 3 in (d). 
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Figure 11: Schematic view of stability analysis for ring network with synaptic 
depression. 
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Figure 12: Solid and dashed lines in (a), (c), and (e) represent stable and unsta- 
ble solutions to the amplitude of localized activity, mi. (b), (d), (e) Distribution 
of eigenvalues for Jacobian matrix H (eq. I60p for perturbations Srhk and SXk 
{—K < k < K ~ I), (b) K = 5 X 10^. The size of the Jacobian matrix H is 
2000 X 2000. (d) if = 5 X 10. The size of the Jacobian matrix H is 200 x 200. 
(e) K = 5. The size of the Jacobian matrix iJ is 20 x 20. 



7 = 2.5, r = 3,f3Ji = 20 




Figure 13: Solid and dashed lines in (a), (c), (e) represent stable and unstable 
solutions to the amplitude of localized activity mi. (b), (d), (e) Distribution 
of eigenvalues for the Jacobian matrix H (eq. I60|) for perturbations 6mk, SXk 
{-K <k<K-l). (h) K = 5x 10^. Size of Jacobian matrix H is 2000 x 2000. 
(d) K = 5 xlO. Size of Jacobian matrix H is 200 x 200. (e) K = 5. Size of 
Jacobian matrix iJ is 20 x 20. 
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